Theoretical Analysis of Bayesian Optimisation with Unknown Gaussian Process Hyper-Parameters

Bayesian optimisation has gained great popularity as a tool for optimising the pa-rameters of machine learning algorithms and models. Somewhat ironically, settingup the hyper-parameters of Bayesian optimisation methods is notoriously hard.While reasonable practical solutions have been advanced, they can often fail tofind the best optima. Surprisingly, there is little theoretical analysis of this crucialproblem in the literature. To address this, we derive a cumulative regret boundfor Bayesian optimisation with Gaussian processes and unknown kernel hyper-parameters in the stochastic setting. The bound, which applies to the expectedimprovement acquisition function and sub-Gaussian observation noise, providesus with guidelines on how to design hyper-parameter estimation methods. A sim-ple simulation demonstrates the importance of following these guidelines.